Search for: All records

Abstract Demand is growing for the dense and high-performing IT computing capacity to support artificial intelligence, deep learning, machine learning, autonomous cars, the Internet of Things, etc. This led to an unprecedented growth in transistor density for high-end CPUs and GPUs, creating thermal design power (TDP) of even more than 700 watts for some of the NVIDIA existing GPUs. Cooling these high TDP chips with air cooling comes with a cost of the higher form factor of servers and noise produced by server fans close to the permissible limit. Direct-to-chip cold plate-based liquid cooling is highly efficient and becoming more reliable as the advancement in technology is taking place. Several components are used in the liquid-cooled data centers for the deployment of cold plate-based direct-to-chip liquid cooling like cooling loops, rack manifolds, CDUs, row manifolds, quick disconnects, flow control valves, etc. Row manifolds used in liquid cooling are used to distribute secondary coolant to the rack manifolds. Characterizing these row manifolds to understand the pressure drops and flow distribution for better data center design and energy efficiency is important. In this paper, the methodology is developed to characterize the row manifolds. Water-based coolant Propylene glycol 25% was used as the coolant for the experiments and experiments were conducted at 21 °C coolant supply temperature. Two, six-port row manifolds' P-Q curves were generated, and the value of supply pressure and the flowrate were measured at each port. The results obtained from the experiments were validated by a technique called flow network modeling (FNM). FNM technique uses the overall flow and thermal characteristics to represent the behavior of individual components. 

Data center cooling systems have undergone a major transformation in the persistent pursuit of better performance and lower energy use. Liquid cooling systems, particularly direct-to-chip systems, have emerged as a promising solution to address the increasing heat dissipation challenges. One critical component of such systems is the filtration mechanism, responsible for safeguarding the integrity and efficiency of the cooling process. These factors are pivotal in ensuring the reliable and sustainable operation of liquid cooling systems in high-demand applications, where electronic components continually push the boundaries of heat generation. This study undertakes a thorough examination of filters of different mesh size used in direct-to-chip liquid cooling systems. The research is multifaceted, encompassing the evaluation of filter performance, pressure drop characteristics, and long-term durability. The methodology employed in this research combines testing with a coolant distribution unit and rack setup to provide a holistic perspective on filter functionality. Findings from this study shed light on the key parameters that influence filter performance. Ultimately, the results of this research promise to contribute significantly to the advancement of direct-to-chip liquid cooling systems, facilitating the continued evolution of electronics in diverse fields, such as high-performance computing, data centers, and emerging technologies. With a focus on enhancing system reliability, efficiency, and sustainability, this study seeks to provide a valuable resource for engineers and researchers in the pursuit of effective cooling solutions for cutting-edge electronic applications. 

Effective cooling is crucial for high-power liquid-cooled servers to ensure optimal performance and reliability ofcomponents. Thermal characterization is necessary to ensure that the cooling system functions as intended, is energy efficient, and minimizes downtime. In this study, a proposed methodology for thermal characterization of a high-powerliquid-cooled server/TTV [server and TTVs (thermal test vehicle) are used interchangeably] is presented. The server layout includes multiple thermal test vehicle setups equipped with direct-to-chip cold plates, with two or more connected in series to form a TTV cooling loop. These cooling loops are connected in parallel to the supply and return plenums of the cooling loop manifold, which includes a chassis-level flow distribution manifold. To obtain accurate measurements, two identical server/TTV prototypes are instrumented with sensors for coolant flow rate and temperature measurements for every TTV cooling loop. Four ultrasonic flow sensors are installed in the flow verification server/TTV to measure the coolant flow rate to each TTV cooling loop. In the thermal verification server, thermistors are installed at the outlet of each GPU heater of TTV cooling loop to log temperature measurements. The amount of heat captured by the coolant in each TTV cooling loop is subsequently estimated based on the flow rates determined from the flow verification server.This methodology enables precise characterization of the thermal performance of high-power liquid-cooled servers,ensuring optimal functionality, energy efficiency, and minimized downtime. 

Abstract In recent years there has been a phenomenal development in cloud computing, networking, virtualization, and storage, which has increased the demand for high performance data centers. The demand for higher CPU (Central Processing Unit) performance and increasing Thermal Design Power (TDP) trends in the industry needs advanced methods of cooling systems that offer high heat transfer capabilities. Maintaining the CPU temperature within the specified limitation with air-cooled servers becomes a challenge after a certain TDP threshold. Among the equipments used in data centers, energy consumption of a cooling system is significantly large and is typically estimated to be over 40% of the total energy consumed. Advancements in Dual In-line Memory Modules (DIMMs) and the CPU compatibility led to overall higher server power consumption. Recent trends show DIMMs consume up to or above 20W each and each CPU can support up to 12 DIMM channels. Therefore, in a data center where high-power dense compute systems are packed together, it demands efficient cooling for the overall server components. In single-phase immersion cooling technology, electronic components or servers are typically submerged in a thermally conductive dielectric fluid allowing it to dissipate heat from all the electronics. The broader focus of this research is to investigate the heat transfer and flow behavior in a 1U air cooled spread core configuration server with heat sinks compared to cold plates attached in series in an immersion environment. Cold plates have extremely low thermal resistance compared to standard air cooled heatsinks. Generally, immersion fluids are dielectric, and fluids used in cold plates are electrically conductive which exposes several problems. In this study, we focus only on understanding the thermal and flow behavior, but it is important to address the challenges associated with it. The coolant used for cold plate is 25% Propylene Glycol water mixture and the fluid used in the tank is a commercially available synthetic dielectric fluid EC-100. A Computational Fluid Dynamics (CFD) model is built in such a way that only the CPUs are cooled using cold plates and the auxiliary electronic components are cooled by the immersion fluid. A baseline CFD model using an air-cooled server with heat sinks is compared to the immersion cold server with cold plates attached to the CPU. The server model has a compact model for cold plate representing thermal resistance and pressure drop. Results of the study discuss the impact on CPU temperatures for various fluid inlet conditions and predict the cooling capability of the integrated cold plate in immersion environment. 

Abstract Data centers are critical to the functioning of modern society as they host digital infrastructure. However, data centers can consume significant amounts of energy, and a substantial amount of this energy goes to cooling systems. Efficient thermal management of information technology equipment is therefore essential and allows the user to obtain peak performance from a system and enables higher equipment reliability. Thermal management of data center electronics is becoming more challenging due to rising power densities at the chip level. Cooling technologies like single-phase immersion cooling allow overcoming many such challenges owing to their higher thermal mass, lower fluid pumping powers, and potential component reliability enhancements. It is known that immersion cooling deployments require extremely low coolant flow rates, and, in many cases, natural convection can also be used to sufficiently dissipate the heat from the hot server components. It, therefore, becomes difficult to ascertain whether the rate of heat transfer is being dominated by forced or natural convection. This may lead to ambiguity in choosing an optimal heat sink solution and a suitable system mechanical design due to unknown flow regimes, further leading to sub-optimal system performance. Mixed convection can be used to enhance heat transfer in immersion cooling systems. The present investigation quantifies the contribution of mixed convection using numerical methods in an immersion-cooled server. An open compute server with dual CPU sockets is modeled on Ansys Icepak with varying power loads of 115W, 160W and 200W. The chosen dielectric fluid for this single-phase immersion-cooled setup is EC-100. Steady-state Computational Fluid Dynamics (CFD) simulations are conducted for forced, natural, and mixed convection heat transfer in a thermally shadowed server configuration at varying inlet flow rates. A baseline heat sink and an optimized heat sink with an increased fin thickness and reduced fin count are utilized for performance comparison. The effect of varying Reynolds number and Richardson number on the heat transfer rate from the heat sink is discussed to assess the flow regime, stability of the flow around the submerged components which depends on the geometry, orientation, fluid properties, flow rate and direction of the flow. The dimensionless numbers’ influence on heat transfer rate from a conventional air-cooled heat sink in immersion versus an immersion-optimized heat sink is also compared. The impact of server orientation on heat transfer behavior for the immersion optimized heat sink is also studied on heat transfer behavior for the immersion optimized heat sink. 

Abstract Direct Liquid Cooling (DLC) has emerged as a promising technology for thermal management of high-performance computing servers, enabling efficient heat dissipation and reliable operation. Thermal performance is governed by several factors, including the coolant physical properties and flow parameters such as coolant inlet temperature and flow rate. The design and development of the coolant distribution manifold to the Information Technology Equipment (ITE) can significantly impact the overall performance of the computing system. This paper aims to investigate the hydraulic characterization and design validation of a rack-level coolant distribution manifold or rack manifold. To achieve this goal, a custom-built high power-density liquid-cooled ITE rack was assembled, and various cooling loops were plugged into the rack manifold to validate its thermal performance. The rack manifold is responsible for distributing the coolant to each of these cooling loops, which is pumped by a CDU (Coolant Distribution Unit). In this study, pressure drop characteristics of the rack manifold were obtained for flow rates that effectively dissipate the heat loads from the ITE. The pressure drop is a critical parameter in the design of the coolant distribution manifold since it influences the flow rate and ultimately the thermal performance of the system. By measuring the pressure drop at various flow rates, the researchers can accurately determine the optimum flow rate for efficient heat dissipation. Furthermore, 1D flow network and CFD models of the rack-level coolant loop, including the rack manifold, were developed, and validated against experimental test data. The validated models provide a useful tool for the design of facility-level modeling of a liquid-cooled data center. The CFD models enable the researchers to simulate the fluid flow and heat transfer within the cooling system accurately. These models can help to design the coolant distribution manifold at facility level. The results of this study demonstrate the importance of the design and development of the coolant distribution manifold in the thermal performance of a liquid-cooled data center. The study also highlights the usefulness of 1D flow network and CFD models for designing and validating liquid-cooled data center cooling systems. In conclusion, the hydraulic characterization and design validation of a rack-level coolant distribution manifold is critical in achieving efficient thermal management of high-performance computing servers. This study presents a comprehensive approach for hydraulic characterization of the coolant distribution manifold, which can significantly impact the overall thermal performance and reliability of the system. The validated models also provide a useful tool for the design of facility-level modeling of a liquid-cooled data center. 

Abstract Due to the increasing computational demand driven by artificial intelligence, machine learning, and the Internet of Things (IoT), there has been an unprecedented growth in transistor density for high-end CPUs and GPUs. This growth has resulted in high thermal dissipation power (TDP) and high heat flux, necessitating the adoption of advanced cooling technologies to minimize thermal resistance and optimize cooling efficiency. Among these technologies, direct-to-chip cold plate-based liquid cooling has emerged as a preferred choice in electronics cooling due to its efficiency and cost-effectiveness. In this context, different types of single-phase liquid coolants, such as propylene glycol (PG), ethylene glycol (EG), DI water, treated water, and nanofluids, have been utilized in the market. These coolants, manufactured by different companies, incorporate various inhibitors and chemicals to enhance long-term performance, prevent biogrowth, and provide corrosion resistance. However, the additives used in these coolants can impact their thermal performance, even when the base coolant is the same. This paper aims to compare these coolant types and evaluate the performance of the same coolant from different vendors. The selection of coolants in this study is based on their performance, compatibility with wetted materials, reliability during extended operation, and environmental impact, following the guidelines set by ASHRAE. To conduct the experiments, a single cold plate-based benchtop setup was constructed, utilizing a thermal test vehicle (TTV), pump, reservoir, flow sensor, pressure sensors, thermocouple, data acquisition units, and heat exchanger. Each coolant was tested using a dedicated cold plate, and thorough cleaning procedures were carried out before each experiment. The experiments were conducted under consistent boundary conditions, with a TTV power of 1000 watts and varying coolant flow rates (ranging from 0.5 lpm to 2 lpm) and supply coolant temperatures (17°C, 25°C, 35°C, and 45°C), simulating warm water cooling. The thermal resistance (Rth) versus flow rate and pressure drop (ΔP) versus flow rate graphs were obtained for each coolant, and the impact of different supply coolant temperatures on pressure drop was characterized. The data collected from this study will be utilized to calculate the Total Cost of Ownership (TCO) in future research, providing insights into the impact of coolant selection at the data center level. There is limited research available on the reliability used in direct-to-chip liquid cooling, and there is currently no standardized methodology for testing their reliability. This study aims to fill this gap by focusing on the reliability of coolants, specifically propylene glycols at concentrations of 25%. To analyze the effectiveness of corrosion inhibitors in these coolants, ASTM standard D1384 apparatus, typically used for testing engine coolant corrosion inhibitors on metal samples in controlled laboratory settings, was employed. The setup involved immersing samples of wetted materials (copper, solder coated brass, brass, steel, cast iron, and cast aluminum) in separate jars containing inhibited propylene glycol solutions from different vendors. This test will determine the reliability difference between the same inhibited solutions from different vendors. 

Abstract The data center’s server power density and heat generation have increased exponentially because of the recent, unparalleled rise in the processing and storing of massive amounts of data on a regular basis. One-third of the overall energy used in conventional air-cooled data centers is directed toward cooling information technology equipment (ITE). The traditional air-cooled data centers must have low air supply temperatures and high air flow rates to support high-performance servers, rendering air cooling inefficient and compelling data center operators to use alternative cooling technology. Due to the direct interaction of dielectric fluids with all the components in the server, single-phase liquid immersion cooling (Sp-LIC) addresses mentioned problems by offering a significantly greater thermal mass and a high percentage of heat dissipation. Sp-LIC is a viable option for hyper-scale, edge, and modular data center applications because, unlike direct-to-chip liquid cooling, it does not call for a complex liquid distribution system configuration and the dielectric liquid can make direct contact with all server components. Immersion cooling is superior to conventional air-cooling technology in terms of thermal energy management however, there have been very few studies on the reliability of such cooling technology. A detailed assessment of the material compatibility of different electronic packaging materials for immersion cooling was required to comprehend their failure modes and reliability. For the mechanical design of electronics, the modulus, and thermal expansion are essential material characteristics. The substrate is a crucial element of an electronic package that has a significant impact on the reliability and failure mechanisms of electronics at both the package and the board level. As per Open Compute Project (OCP) design guidelines for immersion-cooled IT equipment, the traditional material compatibility tests from standards like ASTM 3455 can be used with certain appropriate adjustments. The primary focus of this research is to address two challenges: The first part is to understand the impact of thermal aging on the thermo-mechanical properties of the halogen-free substrate core in the single-phase immersion cooling. Another goal of the study is to comprehend how thermal aging affects the thermo-mechanical characteristics of the substrate core in the air. In this research the substrate core is aged in synthetic hydrocarbon fluid (EC100), Polyalphaolefin 6 (PAO 6), and ambient air for 720 hours each at two different temperatures: 85°C and 125°C and the complex modulus before and after aging are calculated and compared. 

In the United States, out of the total electricity produced, 2% of it is consumed by the data center facility, and up to 40% of its energy is utilized by the cooling infrastructure to cool all the heat-generating components present inside the facility, with recent technological advancement, the trend of power consumption has increased and as a consequence of increased energy consumption is the increase in carbon footprint which is a growing concern in the industry. In air cooling, the high heat- dissipating components present inside a server/hardware must receive efficient airflow for efficient cooling and to direct the air toward the components ducting is provided. In this study, the duct present in the air-cooled server is optimized and vanes are provided to improve the airflow, and side vents are installed over the sides of the server chassis before the duct is placed to bypass some of the cool air which is entering from the front where the hard drives are present. Experiments were conducted on the Cisco C220 air-cooled server with the new duct and the bypass provided, the effects of the new duct and bypass are quantified by comparing the temperature of the components such as the Central Processing Unit (CPUs), and Platform controller hub (PCH) and the savings in terms of total fan power consumption. A 7.5°C drop in temperature is observed and savings of up to 30% in terms of fan power consumption can be achieved with the improved design compared with the standard server.